A multi-layer model for anomaly intrusion detection using program sequences of system calls

In this paper we present a new method to process sequences of system calls for anomaly intrusion detection. The key idea is to build a multi-layer model of program behaviours based on both hidden Markov models and enumerating methods for anomaly intrusion detection, which differs from the conventional single layer approach. Our experiments on Unix sendmail program have shown that the model is better in detecting anomalous behaviour of programs in terms of accuracy and response time. As we use the temporal characteristics in the model, it is suitable for online host-based intrusion detection systems in LAN environment. Index Terms Intrusion detection, anomaly detection, hidden Markov model, machine learning, system call sequence.